Sensor system with active illumination

ABSTRACT

The present invention relates to vision sensors based on an active illumination. An imaging system includes an imaging sensor and is adapted to process an image of a scene being illuminated by at least two different illumination sources each having a wavelength in the near infrared range. In a variant, the imaging system is adapted to use an illumination source having a modulation frequency below the modulation frequency used to perform a three dimensional time of flight measurement. In a variant, the imaging system is adapted to acquire a reduced number of samples per frame than used in time of flight measurements.

FIELD OF THE INVENTION

The present invention relates to vision sensors based on an activeillumination. The invention improves image quality of an active imagingsystem, improves the dynamic range and its background light stability.Such sensors typically are used to detect and measure theirsurroundings, objects or people.

BACKGROUND ART

Active illuminations are implemented in many camera designs to improvethe camera measurement results. Camera is understood as anelectro-optical device containing an image sensor with an array ofpixels.

In many machine vision applications, active illuminations are used toguarantee for certain light intensity level on the camera system. Bydoing so. the performance of an active system becomes less prone tolighting changes in its surrounding. The stability and reliability isimproved. In addition, by adding illumination power to the system,shorter exposures can be set and higher frame rates achieved.

Other sensors with an active illumination camera system use specificproperties of the object of interest in the scene, e.g. an applicationmight use reflectors in the scene that can easily be recognized andpossibly tracked.

Other applications are using specific reflection properties of theobject of interest such as the reflection properties of eyes. Thetypical back-reflection of the eyes captured by a system with activeillumination can be used to detect and track eyes and to count the eyeblinking to e.g. check a driver's drowsiness and build a fatiguemeasurement sensor.

In interferometers, an active system illuminates the object of interestand a reference target. Based on the interference of the reflections,depths can be measured and analyzed.

Other active systems modulate the illumination to get more informationabout its surroundings. Such illuminations are typically modulated intime (temporal modulation) or in space and can be applied to e.g.measure distances to map the environment in 3D. Temporal modulation isimplemented in so-called time-of-flight (TOF) cameras (indirect anddirect); spatial modulation is used in triangulation-based depthmeasurement system, also called structured light technology.

Time of flight image sensor pixels are dedicated pixels designs toguarantee an extremely fast transfer of the photo-generated charges totheir storage nodes. Higher modulation frequencies result in betterdepth noise performance. Therefore, TOF demodulation pixels typicallywork in the range of a few tens of MHz up to a several hundreds of MHz.Further, TOF pixels often include background light suppression onpixel-level, in order to increase the dynamic range of the TOF imagingsystem. In most implementations, TOF pixels include two storage nodes tostore and integrate two sampled signals per pixel.

DISCLOSURE OF THE INVENTION

The robustness of all actively illuminated imaging systems benefit ifbackground light signal such as sun light can be removed. Most of thetime, background light signal cancellation is achieved by theacquisition of two consecutive images, one with illumination turned on,one with the illumination turned off. The subtraction of the two imagesresults in an image including the intensity information of the activeillumination only. The drawback of such an approach is first that thesystem needs to acquire two separate images. The scene or object in thescene might change different from image to image and the backgroundsubtraction in that case not ideal. Further, the two acquired imagesneed to handle the full dynamic signal range from the background lightand the active light. Even though the background light signal is notrequired, it eats up the dynamic range of the system.

It is an object of this invention to provide an imaging system withimproved dynamic range. It is a further objective of this invention toprovide an imaging system for three-dimensional measurement withimproved resolution and less optical power consumption. It is further anobject of this invention to provide a two-dimensional intensity and athree-dimensional imaging system.

According to the present invention, these objectives are achievedparticularly through the features of the independent claims. Inaddition, further advantageous embodiments follow from the dependentclaims and the description.

According to the present invention an imaging system includes an imagingsensor. The imaging system is adapted to process an image of a scenebeing illuminated by at least two different illumination sources eachhaving a wavelength in the near infrared range. Preferably, thewavelength is between 800 and 1000 nm. The illumination sources may beplaced next to the imaging sensor and may be synchronized with the imagesensor. The at least two illumination sources may be placed to minimizeocclusion.

In some embodiments, the wavelengths of the at least two differentillumination sources are different, e.g. at least one illuminationsource is at around 850 nm and at least one illumination source is ataround 940 nm.

In some embodiments, the at least two different illumination sourcesinclude at least one structured illumination source. For example, incase of the image sensor being a time-of-flight image sensor, thisprovides the benefit of the advantages provided by the time of flightmeasurement and the structured illumination source measurement.

In some embodiments, the at least two different illumination sourcesinclude at least one uniform illumination source.

In some embodiments, the at least two different illumination sources areeach in the near infrared range but include at least one uniformillumination source and at least one structured illumination source. Theacquisition of an image based on the structured illumination source maybe interleaved with the acquisitions based on the uniform illuminationsource. The actual raw images based on the structured illuminationsource do not represent its acquired environment in a conventionalcolour or intensity representation. For this reason, state-of-the artsystems based on structured illumination sources add a second imagesensing device, typically an RGB sensor. By implementing two differentillumination sources as proposed in the invention, for example, onestructured illumination source and one uniform illumination source, theimaging system may derive the depth information and generate arepresentative intensity (or black and white) image by the same imagesensor. This further allows an easy and one to one mapping of the 3D maponto the 2D intensity image. In case the image sensor is a TOF imagesensor, the TOF image sensor and the two different illumination sourcescan be further temporally modulated and synchronized.

In some embodiments, the imaging system includes a structuredillumination source and a uniform illumination source, each havingsimilar central wavelengths, both preferably between 800 and 1000 nm. Insuch an embodiment, the image from the structured illumination sourceand the image from the uniform illumination source can each be imagedthroughout the same optical path and a narrow optical bandpass filtercan be implemented in the optical path. The implementation of a narrowbandpass filter allows to optically blocking out as much of thebackground light signal as possible.

In some embodiments, the image sensor is a time of flight sensor.

In some embodiments, the at least two different illumination sourcesinclude at least one illumination source which is temporally modulatedduring exposure. Appropriate temporal modulation scheme enable to reduceartefacts caused by changing scene or moving objects during theacquisition and further enable to avoid interferences from other imagingsystem in the area.

In some embodiments, the at least two different illumination sources areon the same emitting die. This is of special interest, in case the atleast two illuminations sources are structured illumination sources andare both temporally modulated. A system set up with the at least twostructured illumination sources on the same emitting die and modulatethem in sync with the image sensor allows higher information densitylevel on the structured image.

Moreover, the invention is directed to an imaging method using animaging sensor. Images of a scene being illuminated by at least twodifferent illumination sources each having a wavelength in the nearinfrared range is processed. In a variant, the wavelengths of the atleast two different illumination sources are different. In a variant,the at least two different illumination sources include at least onestructured illumination source. In a variant, the at least two differentillumination sources include at least one uniform illumination source.In a variant, the image sensor used is a time of flight sensor. In avariant, the at least two different illumination sources include atleast one uniform illumination source modulated with a modulationfrequency below the modulation frequency used in time of flightmeasurement. In a variant, the at least two different illuminationsources include at least one structured illumination source which istemporally modulated during exposure. In a variant, the at least twodifferent illumination sources are on the same light emitting die.

According to the present invention, an imaging system including a timeof flight sensor is adapted to use an illumination source having amodulation frequency below the modulation frequency used to perform a 3dimensional time of flight measurement.

In preferred specific embodiments, the present invention proposes toimplement in the imaging system a time-of-flight sensors orarchitectures with typically two storage nodes per pixel, preferablyeven containing some sort of in-pixel background suppressing capability.Further, in the actively illuminated imaging system, the time of flightimage sensor and the illumination source are controlled with such lowmodulation frequencies that the actual time-of-flight of the signal hasa negligible effect on the sampling signals. Further, the inventionproposes to implement an acquisition timing on the imaging system thatresults in a number of samples and acquisitions that are not sufficientto perform actual time-of-flight measurements and base the imageevaluation on those acquisitions only. Time-of-flight pixels include inmost practical implementation two storage nodes and capture at least twobut most commonly four consequent but phase delayed images to derivedepth information. In this embodiment, one storage node of the pixel ispreferably used to integrate background light only, which then issubtracted from the other storage node, in which the background lightsignal together with the actively emitted and reflected light isintegrated. The acquisition and integration of the signals, and theirtransfer to the two time-of-flight pixel storage nodes is preferablyrepeated and interleaved many times during an acquisition. The resultingdifferential pixel output will then be a representation of the activelyemitted signal only, which makes the system more robust with respect tochanging lighting conditions in the surroundings.

In some embodiments, the modulation frequency is between 100 Hz and 1MHz. The slower modulation reduces power consumption of the imagingsystem compared to toggling at a few tens of MHz in state-of-the-arttime-of-flight measurement systems. Further, it reduces the speedrequirement of the illumination sources and driver and increases themodulation and demodulation efficiency. Highly efficient high powerlight sources such as high power LEDs that are typically too slow fortime-of-flight measurements can be implemented.

In some embodiments, the imaging system is adapted to perform a directsubtraction on the level of a pixel of the time-of-flight sensor. Theon-pixel background light cancellation of time-of-flight pixels helps inother systems with active illumination sources to avoid saturation dueto background light. In addition, the requirements to the analogue todigital conversion are relaxed since the background level is alreadysubtracted on pixel-level and does not need to be converted anymore.Further, light modulation within a system with active illumination,interleaving the integration of signal light and background light duringthe same frame acquisition, and integrating over several cycles reduceschallenges of motion in the scene, and with appropriate temporalmodulation schemes even enables parallel operation of several systemswith reduced interferences.

In some embodiments, the imaging system is adapted to use a structuredillumination source. The structured illumination source is synchronizedwith the the time-of-flight image sensor and is used to capture theimage based on the structured illumination source and to derive depthinformation.

In some embodiments, the imaging system includes a time-of-flight imagesensor and at least one illumination source. The time-of-flight imagesensor is used to sample the back-reflected light. The evaluation of thesignal is based on a number of acquisitions and acquired samples perframe, which are not sufficient to derive depth information based on thetime-of-flight principle.

In some embodiments, the imaging system is adapted to use apseudo-random modulated illumination source, so that differentinterferences between different acquisition systems are minimized.

In some embodiments, the imaging system is adapted to use at least twoillumination sources having at least two different wavelengths. By doinga differential readout or on-pixel subtraction, the difference image ofthe two illumination sources can directly be measured. This enables, forexample, to build a highly robust eye tracking system.

In some embodiments, the present invention provides an imaging systemwhich uses a time-of-flight sensor, wherein an illumination source has amodulation frequency below the modulation frequency used to perform athree dimensional time of flight measurement.

In some embodiments, the imaging system is adapted to acquire a reducednumber of samples per frame than required to perform time of flightmeasurements.

In some embodiments, the timing of an imaging system including atime-of-flight sensor and at least one illumination source is adapted toacquire a reduced number of samples per frame than required to derivetime of flight measurements.

Moreover, the invention is directed to an imaging method using atime-of-flight sensor. An illumination source having a modulationfrequency below the modulation frequency used in time of flightmeasurements is used. In a variant, the modulation frequency used isbetween 100 Hz and 1 MHz. In a variant, a direct subtraction on thelevel of a pixel of the time-of-flight sensor is performed. In avariant, a structured illumination source is used. In a variant, twoillumination sources having at least two different wavelengths are used.In a variant, a pseudo-random temporally modulated illumination sourceis used. In a variant, a reduced number of samples per frame than usedin time of flight measurements is acquired.

According to the present invention, the imaging system is adapted toacquire a reduced number of samples per frame than used in time offlight measurements.

In some embodiments, the imaging system is adapted to use anillumination source having a modulation frequency below the modulationfrequency used in time of flight measurements, in particular amodulation frequency between 100 Hz and 1 MHz.

In some embodiments, the imaging system is adapted to perform a directsubtraction on the level of a pixel of the time-of-flight sensor.

In some embodiment, the imaging system is adapted to use a structuredillumination source.

In some embodiments, the imaging system is adapted to use at least twoillumination sources having at least two different wavelengths.

In some embodiments, the imaging system is adapted to use apseudo-random temporally modulated illumination source.

Moreover, the invention is directed to an imaging method using atime-of-flight sensor. A reduced number of samples per frame than usedin time of flight measurements is acquired. In a variant, anillumination source having a modulation frequency below the modulationfrequency used in time of flight measurements is used, in particular amodulation frequency between 100 Hz and 1 MHz. In a variant, a directsubtraction on the level of a pixel of the time-of-flight sensor isperformed. In a variant, a structured illumination source is used. In avariant, two illumination sources having at least two differentwavelengths are used. In a variant, a pseudo-random temporally modulatedillumination source is used.

BRIEF DESCRIPTION OF THE DRAWINGS

The herein described invention will be more fully understood from thedetailed description given herein below and the accompanying drawingswhich should not be considered limiting to the invention described inthe appended claims. The drawings are showing:

FIG. 1 a) shows a diagram of the state-of-the-art TOF pixel with thedifferent building blocks. b) to d) plot prior art of timing diagramsand samplings to derive depth measurement based on the time-of-flightprinciple;

FIG. 2 shows the most common used prior art acquisition timing of thefour samples required to derive depth measurement on time-of-flight 3dimensional imaging systems;

FIG. 3 a) illustrates an imaging system 100 according to the inventionwith the time-of-flight image sensor 110 and the active illuminationsource 120, both controlled and synchronized by the controller 140, b)an embodiment of a rough timing diagram according to the invention andc) an embodiment of a more detailed timing diagram according to theinvention;

FIG. 4 shows a imaging system 100 according to the invention based on astructured illumination source 121;

FIG. 5 illustrates an imaging system 100 according to the invention foreye/pupil detection applications based on two illumination sources 122and 123 with different wavelengths in the near infrared range and atime-of-flight image sensor (110);

FIG. 6 shows a imaging system 105 according to the invention, thatincludes a first structured illumination source 121 and a seconddifferently structured illumination source 125;

FIG. 7 shows an embodiment of an imaging system according to theinvention, which enables to gather three-dimensional and colour orgreyscale information from the scene/object through the same opticalpath and image sensor. a) illustrates the operation of the acquisitionto acquire depth information. A sample greyscale image captured with astructured illumination source is shown in c). b) plots the imagingsystem and its operation during the acquisition of the greyscale imageand the resulting greyscale image is plotted in d); and

FIG. 8 shows an actual design according to the invention.

MODE(S) FOR CARRYING OUT THE INVENTION

Time-of-flight imaging systems area capable to determine the time ittakes to the emitted light to travel from the measurement system to theobject and back. The emitted light signal is reflected by an object at acertain distance. This distance corresponds to a time delay from theemitted signal to the received signal caused by the travel time of thelight from the measurement system to the object and back. In indirecttime-of-flight measurements, the distance-dependant time delaycorresponds to a phase delay of the emitted to the received signals.Further, the received signal not only includes the back-reflectedemitted signal, but may also have background light signal from e.g. thesun or other light sources. A state-of-the-art time-of-flight pixel isillustrated in FIG. 1a . The time-of-flight pixel includes aphoto-sensitive area P, connected with a first switch SW1 to storagenode C1 and connected with a second switch SW2 to storage node C2. Thesampling of the photo-generated electrons is done by closing switch SW1and opening SW2 or vice versa. The switches SW1 and SW2 are synchronizedto an illumination source by a controller. In order to enable reasonabletime of flight measurements, the switches SW1 and SW2 and illuminationsources need to operate in the range of around 10 MHz to above 200 MHzand the photo-generated charges have to be transferred from thephoto-sensitive area P to either storage node C1 or C2 within a few ns.Time-of-flight pixels are specifically designed to reach such high-speedsamplings. Possible implementations of such high-speed time-of-flightpixel architectures are described in patents U.S. Pat. No. 5,856,667,EP100998481, EP1513202B1 or U.S. Pat. No. 7,884,310B2. The outputcircuitry C of the time-of-flight pixel typically includes a readoutamplifier and a reset node. In many time-of-flight pixelimplementations, the in-pixel output circuitry C further includes acommon level removal or background subtraction circuitry of the twosamples in the storage nodes C1 and C2. Such in-pixel common signallevel removal drastically increases the dynamic range of time-of-flightpixels. Possible implementations that perform a common level removal ofthe samples while integrating are presented in PCT publicationWO2009135952A2 and in U.S. Pat. No. 7,574,190B2. An approach of anothercommon signal level subtraction of the samples done after the exposureduring the readout cycle of the data is described in U.S. Pat. No.7,897,928B2.

Commercially available 3 dimensional time-of-flight measurement systemsare either based on sine wave or pseudo-noise modulation. Both requireat least three samples to derive phase or depth informationrespectively, offset, and amplitude information. For designsimplification and signal robustness reasons, time-of-flight systemscommonly sample four times the impinging light signal. However, the mostsensitive and therefore most widely used time-of-flight pixelarchitecture includes only two storage nodes, as sketched in FIG. 1a .In order to get the four samples, the system then requires theacquisition of at least two subsequent images. The timing diagram of thesine wave sampling is sketched in FIG. 1b . This received light signalgenerates on the pixel from FIG. 1a a photo-current, which is thensampled and integrated during a first exposure E1, as represented inFIG. 1b . The first exposure E1 is followed by a readout RO1. A secondexposure E2 acquires the samples delayed by 90° compared to E1. Afterthe exposure E2, the newly acquired samples are readout RO2. At thispoint in time D, all data is ready to enable to determine phase, offsetand amplitude information of the received signal. A time zoom in thesampling procedure of E1 is sketched in FIG. 1c , a time zoom intoexposure E2 in FIG. 1d , respectively. The sampling duration is assumedto be half of the period and is integrated over many thousands up tomillions of periods. During such a first exposure E1, the samples at 0°and 180° are directed by the switches SW1 and SW2 to the storage nodesC1 and C2, respectively, as sketched in FIG. 1c . A zoom into the secondexposure E2 is sketched in FIG. 1d . The second exposure E2 has thesamplings delayed by 90° compared to the first exposure E1. Samples 90°and 270° are again integrated in storage node C1, C2. Integration timeof exposure E1 and exposure E2 are the same. At point in time D, whenall four samples have been measured and are available, the phase,amplitude and offset information can be calculated, whereas the phaseinformation directly corresponds to the distance information of themeasured object.

However, due to mismatches, most implementations actually acquire fourinstead of only two images, as proposed in U.S. Pat. No. 7,462,808B2 andsketched in FIG. 2. The first exposure E1 and the second exposure E2 andtheir readouts RO1 and RO2 are performed as described in FIG. 1b butthose exposures are followed by two more exposures E3 and E4 andreadouts RO3 and R04. Exposure E3 acquires the samplings from firstexposure E1 but delayed by 180°, and the exposure E4 corresponds to a180° phase delay of exposure E2. At point in time D, all four samplesare available to calculate phase (or depth respectively), offset andamplitude information of the received signal. As described in U.S. Pat.No. 7,462,808B2, this approach enables to compensate for e.g. pixelcircuitry and response mismatches or driving non-symmetries.

A first general embodiment according to the invention based on animaging system 100 is depicted in FIG. 3a , its rough timing on FIG. 3band a more detailed timing in FIG. 3c . The imaging system includes atime-of-flight image sensor 110 consisting of time-of-flight pixels asillustrated in FIG. 1a , an illumination source 120, an optical system130 and a controller 140. The emitted light 120 a is reflected by theobject 10. The back-reflected light 120 b is imaged by the opticalsystem 130 onto the time-of-flight image sensor 110. The time-of-flightimage sensor 110 and the illumination source 120 are temporallymodulated and synchronized by the controller 140 such that one storagenode of the time-of-flight pixels on the time-of-flight image sensor 110integrate all photo-generated electrons while the illumination source120 is turned on, and a second storage node collects all photo-generatedelectrons when the illumination source 120 is turned off. This on/offcycle might be repeated many times. The present invention proposes toapply such low modulation frequencies to the high-speed time-of-flightimage sensor and pixels that the actual impact of the time of flight ofthe emitted and back-reflected light is negligible and all emitted lightis preferably captured into a single storage node of each pixel on thetime-of-flight image sensor. Further, as depicted in FIG. 3b , it isproposed to only capture a reduced number of samples and acquisitions,which does not enable to derive actual time-of-flight measurements. Afirst exposure E is followed by the readout RO. During the exposure E,photo-generated charges are either transferred to storage node C1 orstorage node C2, synchronized with the illumination source 120. In thegiven embodiment, one single acquisition rather than the at least two orfour acquisitions as required to gather all necessary (at least three)samples to derive TOF information is proposed. At the point in time D,the number of samples does not allow to deduce time-of-flightinformation of the captured signal. On that single exposure example asvariant of the invention, differential imaging of the two samples iscarried out. The modulation can be done in a pseudo-random way as hintedin FIG. 3a . By doing so, disturbing interferences between differentimaging systems 100 can be minimized. Next to pseudo-random coding,other known techniques such as phase hopping or frequency hopping,chirping or other division multiple access approaches can be implementedto minimize interferences of the systems.

FIG. 3c shows the timing diagram of the single exposure as sketched inFIG. 3b in more detail, including a circuitry subtracting the twostorage nodes C1 and C2 from each other during the integration. Fdescribes the total frame, E is the actual exposure, RO the readout timeand RS the reset time. The modulation frequency on the present timing ofthe embodiment is much lower than required to do reasonabletime-of-flight imaging. The light L1 is emitted by the illuminationsource 120 and synchronized with the time-of-flight image sensor 110,such that during the “light on” time all photo-generated charges aretransferred to the first storage node C1, while all during the“light-off”-time photo-generated charges are transferred to the secondstorage node C2. The back-reflected and received light signal over timet is plotted as L2 and has some background light component LBG. S1illustrates the integration on the first storage node C1 during the“light on time”, S2 shows the integration on storage node C2 during the“light-off” time. Sdiff shows the signal difference, when actualin-pixel background removal is implemented. In-pixel circuitry C buildsthe difference and, by doing so, removes the background light signal. Inthe illustrated example, the time-of-flight pixel on the time-of-flightimage sensor 110 performs a direct subtraction of the two nodes duringintegration on Sdiff. In other implementations, the time-of-flight pixelperforms the subtraction at the end of the integration time. Bothimplementations increase the dynamic range of the active imaging system.The “light-on” and “light-off” times are repeated many times during theexposure E. Modulation frequencies in the range of hundreds of Hz tobelow MHz enable to reduce the impact of the arrival time of the lightpulses. The total integration time into the first storage node C1 andinto the second storage node C2 of the time-of-flight pixel arepreferably equivalent during the exposure, in order to integrate thesame amount of background light to the two storage nodes on thetime-of-flight image sensor's 110 pixels and to subtract the samebackground or common mode level on those samples. If the two exposuretimes are kept the same, the light pulses might even be shorter than thesample duration into its storage node. This further ensures that thereis no impact on the arrival time of the pulse. In any case, to have bestpossible background light removal, the total exposure time into the twostorage nodes should be the same. After the exposure E, the pixel valuesare read out RO and transferred to the controller 140. Before startingthe next acquisition, the pixels are typically reset RS. In order toreduce mismatches in the photo-response on the two storage nodes, asecond exposure with inverted switching compared to the first exposureE, and subtracting the two images may be beneficial depending on theimage sensor pixel's photo-response. However, the number of acquiredsamples would still not suffice to derive time-of-flight information.

FIG. 4 shows an imaging system 100 according to the invention. Theimaging system 100 includes a time-of-flight image sensor 110 thatconsists of time-of-flight pixels as sketched in FIG. 1a , an opticalsystem 130, a structured illumination source 121 and a controller 140.The controller 140 further synchronizes and temporally modulates thetime-of-flight image sensor 110 and the structured illumination source121. The structured illumination source 121 emits the temporallymodulated light 121 a, which is reflected by the object 10. Theback-reflected light 121 b by the scene 10 is projected by an opticalsystem 130 on the time-of-flight image sensor 110, on which thetime-of-flight pixels demodulate the incoming signal into two storagenodes. The timing of the imaging system 100 can be similar to the onedescribed in FIG. 3. The structured illumination source and the 3dimensional mapping techniques can be applied as presented by the PCTpublication WO2007/105205A2. For the structured illumination source,projection techniques based on speckles caused by interference asdescribed in WO2007/105205A2 might be used, however, other patternprojection techniques might be applied to structured illumination source121, e.g. with refraction based optics or pattern generating masks.Preferably, a random dot pattern is projected. EP2519001A2 teaches theuse of a structured light system based on a sensor with specificallydesigned sensor pixels with transfer gates to first and second storagenodes and using standard slow photodiodes. The present invention incontrary proposes not to apply a specific pixel design, but rather usean existing high-speed time-of-flight image sensor 110 and pixels, applysuch low modulation frequency by the controller 140 on the structuredillumination source 121 and the time-of-flight image sensor 110, thatthe impact of the actual travel time of the light (time-of-flight) isnegligible, and perform the sampling and storage of the back-reflectedlight 121 b on the time-of-flight image sensor 110. Further, thetime-of-flight image sensor 110 is proposed to operate in a mode, thatthe at least three required samples are not captured, but the reducednumber of samples from the time-of-flight image sensor 110 areevaluated. The typical 90° phase shift as it is required for thetime-of-flight imaging system is abandoned and only the differentialimage resulting from the two storage nodes on all pixels on thetime-of-flight image sensor 110, is evaluated. All afore referencedhigh-speed time-of-flight image sensors are capable to demodulate thereflected light temporally into at least two storage nodes at such lowfrequencies unusable for actual time-of-flight measurement system.Further, all of the afore-referenced high-speed time-of-flight imagesensors allow the acquisition of a reduced number of samples thanrequired for time-of-flight image sensing. Therefore all those referencepixel designs can be integrated just as time-of-flight image sensoraccording to the invention. With appropriate in-pixel circuitry, thetime-of-flight image sensor 110 can subtract a common signal level fromthe samples, or simply subtract the two storage node samples from eachother. On the one hand, this increases the dynamic range of the imagingsystem and on the other hand, simplifies the correlation algorithms forthe depth mapping since the dynamic background light signals are removedin the image. Further, by applying appropriate temporal modulationschemes such as code or frequency division multiple accesses, disturbinginterferences of different structured light 3 dimensional imagingsystems can be avoided.

Ideally, the time-of-flight pixels on the time-of-flight image sensor110 transfer charges during the “light-on” time into a first storagenode and charges during the “light-off” time into a second node andperform the subtraction. The temporal modulation and the backgroundcancellation circuit of the time-of-flight pixels add to the structuredlight system all the advantages described above. The modulationfrequency is preferably in the range of a few 100 Hz up to 1 MHz, toolow for actual TOF measurements. Such an imaging system 100 as describedin FIG. 4, including a structured illumination source 121 and atime-of-flight sensor 100, both operated at modulation frequencies farbelow modulation frequencies required for time-of-flight imaging, willdrastically increase the dynamic range of existing 3 dimensional imagingsystem as described in the PCT publications WO2007/105205A2 and/orW2014/014341A1.

FIG. 5 represents an imaging system 100 according to the invention, usedas fatigue sensor. The present embodiment enables to build a highlyrobust eye tracking system, which can be used e.g. for driver'sdrowsiness measurement or fatigue estimation. The imaging system 100according to the invention includes two different illumination sources,each having a wavelength in the near infrared range, whereas a firstillumination source of a first wavelength 122, is at e.g. around 850 nm,and a second illumination source of a second wavelength 123, is at e.g.around 940 nm. It further includes a time-of-flight image sensor 110with time-of-flight pixels according to FIG. 1a , an optical system 130and a controller 140. Both illumination sources 122 and 123 aresynchronized with the time-of-flight image sensor 110 by the controller140. During the exposure, when the first illumination source of a firstwavelength 122 is turned on, the second illumination source of a secondwavelength 123 is turned off and vice versa. The first illuminationsource of a first wavelength 122 emits the light 122 a, the secondillumination source of a second wavelength 123 emit the light 123 a. Theback-reflected light 122 b of the first illumination source of a firstwavelength 122 will be imaged by the optical system 130 on thetime-of-flight pixels of the time-of-flight image sensor 110 andtransferred to the first storage nodes on the time-of-flight pixels,while the back-reflected light 123 b from the illumination source of asecond wavelength 123 will be captured by the same time-of-flight imagesensor 110 and transferred into the time-of-flight pixels' secondstorage nodes. After the exposure and by doing a differential readout oron-pixel signal subtraction, the difference image of the twoillumination sources can directly be measured. Since the retina of ahuman eye still shows direct reflection at around 850 nm and stronglyreduced reflection at around 940 nm, the pupils will clearly be seen inthe difference image and can be tracked easily and robustly. The pupilcan easily be identified in the differential image and the opening andclosing of the eyes be detected. Based on the blinking, the PERCLOSvalue (proportion/percentage of time in a minute that the eye is 80%closed) or a drowsiness factor of e.g. a bus driver can be measured andcorresponding actions initiated. State-of-the-art methods to determinethe drivers drowsiness are mainly based on standard imaging and heavyimage processing as described by Hammoud et al. in U.S. Pat. No.7,253,739B2.

The embodiment of the imaging system 105 sketched in FIG. 6 illustratesa 3 dimensional imaging system according to the invention. The imagingsystem 105 includes a first structured illumination source 121 and asecond structured illumination source 125 of a different type ofstructure. Both illumination sources are at near infrared wavelength.Further, the imaging system includes an image sensor 115, an opticalsystem 130 and a controller 140. The system 105 is setup using a firststructured illumination source 121 and a second structured illuminationsource 125, and both illumination sources are placed next to the imagesensor 115. A controller 140 synchronizes the structured illuminationsources 121, 125 and the image sensor 115. The two illumination sources121 and 125 are preferably placed such as to minimize occlusion. Theemitted light 121 a from the first structured illumination source 121 isreflected and the back-reflected light 121 b is projected by the opticalsystem 130 on the image sensor 115. The emitted light 125 a from thesecond structured illumination source 125 is reflected, too, and theback-reflected light 125 b is projected by the same optical system 130on the same image sensor 115, too. During the acquisition, theillumination sources 121 and 125 may be temporally modulated and in thatcase, are preferably inverted during exposure such that all light fromthe first structured illumination source 121 can be collected in a firststorage node of the pixel on the image sensor 115, and back-reflectedlight 125 b from second structured illumination source 125 is collectedin a second storage node of the pixels on the image sensor 115. Temporalmodulation such as square wave or pseudo noise or others can beimagined. In case the pixels on the image sensor 115 include backgroundsubtraction circuitry, the signal from the first structured illuminationsource 121 will be positive; the signal from the second structuredillumination source 125 will be negative. The drawback of this approachand having the two illumination sources physically separated buttemporally interleaved is that pixels including signal from the firststructured illumination source 121 and the second structuredillumination source 125 may cancel each other. Therefore, one mightthink to have the first structured illumination source 121 e.g. with arandom speckle pattern and the second structured illumination source 125with a stripe shape pattern. Further, it is imaginable that bothstructured illumination sources are integrated as structured patterns onthe same emitting die e.g. a VCSEL array. The VCSEL array may consistsof a first group of emitting laser diodes corresponding to the firststructured illumination source 121 that can be controlled differentlythan a second group of emitting laser diodes corresponding to the secondstructured illumination source 125, both groups are on the same VCSELarray die. An array of VCSEL on the same die, used as projector for animaging system based on a structured illumination source, with the fullarray controlled by a single driving signal is presented in publicationUS2013038881A1. One can imagine to have the first group of laser diodeemitting spots randomly placed on the die, whereas the second group oflaser diode emitting spots from the same die and have the same patternbut are all slight shifted. By projecting this VCSEL array pattern thespots projected from the first group of laser diodes or first structuredillumination source 121 and the spots projected from the second group oflaser diodes or second structured illumination source 125 would notinterfere with each other in space, since their spots are physicallyseparated on the VCSEL array die, and their emitted light is projectedin space by the same projection optics. Further, the two groups oflasers diodes may be controlled in such a way that all back-reflectedlight 121 b of the first structured illumination source 121 is stored ona first storage node of the pixels on the image sensor 115 and allback-reflected light 125 b from the second structured illuminationsource 125 is stored on the second storage node of the pixels on theimage sensor 115. Further, the pixels on the image sensor 115 mayinclude a background cancelling circuitry to enhance the dynamic range.Further, the image sensor 115 may be a time-of-flight image sensor 110.

FIG. 7a to d illustrates an embodiment according to the invention of animaging system 105 including a structured illumination source 121, auniform illumination source 126, an optical system 130, an image sensor115 and a controller 140. Both illumination sources have a wavelength inthe near infrared range. In a first exposure, illustrated in FIG. 7a ,the structured illumination source 121 is turned on, while the uniformillumination source is turned off. As hinted by the drawing in FIG. 7a ,the structured illumination source 121 may be modulated and synchronizedby the controller 140 with the image sensor 115. The emitted light 121 afrom the structured illumination source 121 reaches an object 10 and isreflected. The back-reflected light 121 b form the structuredillumination source 121 is imaged by the optical system 130 onto theimage sensor 115. The image from the structured illumination source 121captured by the images sensor 115 can be used in a traditional way toderive depth information from the scene. However, the acquired greyscaleimage delivered by the image sensor 115 based on the structuredillumination source 121 is not representative and objects and detailsare only vaguely recognizable in many cases. Such a greyscale imagebased on a structured illumination source 121 with a random dot patternis illustrated in FIG. 7c . The loss of details is clearly visible. Tostill be able to have a conventional image, state-of-the-art imagingsystems based on structured illumination sources add next to the firstimage sensor 115 a second image sensing device with its own optical pathto the image sensor. The second image sensor typically is an RGB sensorand delivers a completely independent colour image. However, a secondimage acquired through a different optical path is first of all costlyand second results in many challenges in mapping the two images. Anembodiment of the invented method proposes to acquire a second imagewith the imaging system 105, having a uniform illumination source 126turned on and using the same optical system 130 and image sensor 115.The emitted light 126 a from a uniform illumination source 126 reachesan object 10 in the scene and is reflected. The back-reflected light 126b from the uniform illumination source 126 is imaged by the opticalsystem 130 on the image sensor 115. The operation of the second imageacquisition is depicted in FIG. 7b . As hinted in the drawing in FIG. 7b, the uniform illumination source 126 and the image sensor 115 may besynchronized and modulated by the controller 140. A resulting greyscaleimage of this second image based on the uniform illumination source 126is presented in FIG. 7d . Details are now much better visible, mappingthe 3 dimensional image from the structured illumination source 121 andthe greyscale image from the uniform illumination source 126 is simple.

In order to implement an appropriate optical bandpass filter in theoptical system 130, it is advantageous to have similar wavelengths forthe structured illumination source 121 and the uniform illuminationsource 126. Further, the structured illumination source 121 and theuniform illumination source 126 may be synchronized and modulated withthe image sensor 115 by the controller 140. The image sensor 115 mayfurther have in-pixel background cancelling circuitry in order toincrease the dynamic range.

Further, the image sensor 115 may be a time-of-flight image sensor 110.

FIG. 8 shows an actual embodiment according to the invention. Theimaging system 105 includes an optical system 130, a structuredillumination source 121, and a uniform illumination source 126, eachillumination source having a wave-length in the near infrared range. Theimage sensor 115 behind the optical system and the controller 140 arehidden behind the housing of the imaging system 105 and therefore cannotbe seen in the drawing. In the given example in FIG. 8, the uniformillumination source 126 consists of two equal illumination sources,placed next to the structured illumination source 126, which is centred.The image captured by the image sensor 115 based on the uniformillumination source 126, can be used as representative greyscaleillustration of the scene, while the image captured by the image sensor115 based on the structured illumination source 121 can be used toderive depth information based on the triangulation principle. Thewavelength of the structured illumination source 121 and the uniformillumination source 126 are preferably similar to enable theimplementation of a narrow bandpass filter in the optical system 130 tocut off as much of the background light as possible while passing asmuch of the light from the structured illumination source 121 and theuniform illumination source 126 as possible onto the image sensor 115.Further, the image sensor 115 and the structured illumination source 121may be synchronized and modulated as explained in FIG. 7. Further, theuniform illumination source 126 and the image sensor 115 may besynchronised and modulated as explained in FIG. 7. Further, the imagesensor 115 might be a time-of-flight image sensor 110 and preferably,includes an in-pixel background cancellation circuitry. Based on theimage based on the uniform illumination source 126, appropriate imageprocessing algorithms may improve the depth map reconstructed by theevaluation of the image based on the structured illumination source 121.

In all of the afore-mentioned implementations with modulatedillumination source, the light source pulses have preferably the same orshorter duration than the samplings to one of the storage nodes.Further, the sampling duration to the storage nodes is preferably equalto the two nodes, in order to have equal background light in bothstorage nodes that is cancelled.

The structured illumination source is understood as a spatiallymodulated light source, emitting a fixed pattern. Possible patterns arespeckle patterns (random, pseudo-random or regular), stripe patterns,random binary patterns, etc.

The invention claimed is:
 1. An imaging system comprising: a time offlight imaging sensor; first and second different illumination sourceseach having a respective wavelength in the near infrared range; and acontroller operable to apply a modulation frequency to the time offlight imaging sensor and to the first and second illumination sourcessuch that the time of flight imaging sensor and the first and secondillumination sources are synchronized with one another, and such thatduring an exposure, when the first illumination source is on, the secondillumination source is off, and when the second illumination source ison, the first illumination source is off, the imaging system beingoperable to process an image of a scene being illuminated by the firstand second illumination sources.
 2. The imaging system according toclaim 1, wherein the wavelengths of the first and second illuminationsources differ from one another.
 3. The imaging system according toclaim 1, wherein at least one of the illumination sources is astructured illumination source.
 4. The imaging system according to claim1, wherein at least one of the illumination sources is a uniformillumination source.
 5. The imaging system according to claim 1, whereinat least one of the illumination sources is a illumination source whichis temporally modulated.
 6. The imaging system according to claim 1,wherein the first and second different illumination sources are on thesame light emitting die.
 7. The imaging system according to claim 1,wherein the imaging sensor is a time of flight sensor.
 8. An imagingsystem comprising: a time of flight sensor having a pixel architecturedesigned to handle operation at 10 MHz or more; an illumination source;a controller operable to apply a modulation frequency in a range of 100Hz to 1 MHz to the time of flight sensor and to the illumination sourcesuch that the time of flight sensor and the illumination source aresynchronized with one another, the controller further operable to sampleoutput signals from the time of flight sensor based on light produced bythe illumination source and reflected by an object.
 9. The imagingsystem according to claim 8, wherein the imaging system is operable toperform a direct subtraction at a pixel level of the time of flightsensor.
 10. The imaging system according to claim 8, wherein theillumination source includes a structured illumination source.
 11. Theimaging system according to claim 8 comprising at least two illuminationsources having different wavelengths from one another, wherein thecontroller is operable to synchronize each of the illumination sourceswith the time of flight sensor.
 12. The imaging system according toclaim 8, illumination source includes a pseudo-random modulatedillumination source.
 13. The imaging system according to claim 1 whereinthe first illumination source is operable to produce a random specklepattern, and the second illumination source is operable to produce astripe shape pattern.
 14. The imaging system according to claim 1wherein the a time of flight imaging sensor includes a plurality ofpixels each of which has a respective first and second storage node,wherein the imaging system is operable such that light produced by thefirst illumination source, reflected by an object and detected by aparticular pixel is transferred to the pixel's first storage node, andsuch that light produced by the second illumination source, reflected bythe object and detected by the particular pixel is transferred to thepixel's second storage node.